Polarized illumination and detection for depth sensing

ABSTRACT

A depth camera assembly (DCA) for depth sensing of a local area. The DCA includes a polarized light generator, an imaging device, and a controller. The polarized light generator modulates one or more optical beams emitted from an illumination source to generate modulated light, and projects the modulated light into the local area as polarized light having a first polarization. The imaging device receives light from the local area, the received light including ambient light and a portion of the polarized light reflected from the local area. The imaging device reduces an intensity of the received light having polarization different from a second polarization to generate filtered light substantially composed of light of the second polarization, and detects the portion of the polarized light having the second polarization using the filtered light. The controller determines depth information for the local area based on the detected portion of the polarized light.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/917,705, filed Jun. 30, 2020, which is a continuation of U.S.application Ser. No. 16/201,391, filed Nov. 27, 2018, now U.S. Pat. No.10,740,915, which is a continuation of U.S. application Ser. No.15/636,398, filed Jun. 28, 2017, now U.S. Pat. No. 10,181,200, all ofwhich are incorporated by reference in their entirety.

BACKGROUND

The present disclosure generally relates to depth sensing, andspecifically relates to circularly polarized illumination and detectionfor depth sensing.

To achieve compelling augmented reality (AR) and virtual reality (VR)user experiences, it is desired to create a depth sensing device thatcan determine a dense three-dimensional mapping in both indoor andoutdoor surroundings. A depth camera usually involves structured lightillumination, which is a triangulation technique that makes use of anactive illumination source to project known patterns into a scene. Thedepth camera typically utilizes a two-dimensional pixel array detectorto measure and record light back-scattered from one or more objects inthe scene. Other methods for depth sensing are based on a time-of-flighttechnique, which measures a round trip travel time-of-light projectedinto the scene and returning to pixels on a sensor array. The problemrelated to the depth sensing methods based on structured lightillumination and time-of-flight is related to designing a compact andefficient depth camera that can produce quality depth maps in bothindoor and outdoor environments where background ambient light canstrongly interfere depth measurements. The depth map obtained in theseenvironments typically have large depth errors and a low level ofsignal-to-noise ratio (SNR) due to the strong background ambient light.

SUMMARY

A depth camera assembly (DCA) determines depth information associatedwith one or more objects in a local area. The DCA comprises a polarizedlight generator, an imaging device and a controller.

The polarized light generator is configured to illuminate the local areawith polarized light in accordance with emission instructions. Thepolarized light generator comprises an illumination source, a modulator,and a projection assembly. The illumination source is configured to emitone or more optical beams. In some embodiments, the illumination sourcedirectly generates the one or more optical beams as light of a certainpolarization, e.g., based on a polarizing element integrated into theillumination source or placed in front of the illumination source. Themodulator is configured to modulate the one or more optical beams basedin part on the emission instructions to form a modulated light forscanning the local area. In some embodiments, the modulator isconfigured as a diffractive optical element that diffracts the one ormore optical beams based in part on the emission instructions togenerate a diffracted light for scanning the local area with a widefield-of-view. In some embodiments, the modulator includes a polarizingelement for generating the modulated light as the polarized light usingthe one or more optical beams, based in part on the emissioninstructions. In some embodiments, the modulator configured as anacousto-optic deflector directly generates the modulated light as thepolarized light without any additional polarizing element. In alternateembodiments, the modulator can be configured as a liquid crystal gratingdevice that directly generates the modulated light as the polarizedlight without any additional polarizing element. The projection assemblyis configured to project the polarized light into the local area. Insome embodiments, the projection assembly includes the polarizingelement for generating the polarized light using the modulated lightgenerated by the modulator as un-polarized light. The polarized lightprojected into the local area has a first polarization, e.g., circularpolarization.

The imaging device is configured to capture portions of the polarizedlight reflected from the one or more objects in the local area. Theimaging device includes another polarizing element and a detector. Theother polarizing element of the imaging device is configured to receivelight from the local area, the received light including ambient lightand a portion of the polarized light reflected from the one or moreobjects in the local area. The other polarizing element of the imagingdevice is also configured to reduce an intensity of the received lighthaving polarization different from a second polarization (e.g., theambient light) to generate filtered light substantially composed oflight of the second polarization. In some embodiments, the polarizingelement blocks the ambient light for reaching the detector. The detectoris configured to detect the portion of the polarized light reflectedfrom the one or more objects in the local area having the secondpolarization using the filtered light. The controller may be coupled toboth the polarized light generator and the imaging device. Thecontroller generates the emission instructions and provides the emissioninstructions to the polarized light generator. The controller is alsoconfigured to determine depth information for the one or more objectsbased at least in part on the detected portion of the reflectedpolarized light.

A head-mounted display (HMD) can further integrate the DCA. The HMDfurther includes an electronic display and an optical assembly. The HMDmay be, e.g., a virtual reality (VR) system, an augmented reality (AR)system, a mixed reality (MR) system, or some combination thereof. Theelectronic display is configured to emit image light. The opticalassembly is configured to direct the image light to an exit pupil of theHMD corresponding to a location of a user's eye, the image lightcomprising the depth information of the one or more objects in the localarea determined by the DCA.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a head-mounted display (HMD), in accordance withan embodiment.

FIG. 2 is a cross section of a front rigid body of the HMD in FIG. 1, inaccordance with an embodiment.

FIG. 3 is an example depth camera assembly (DCA), in accordance with anembodiment.

FIG. 4 is a flow chart illustrating a process of polarized illuminationand detection for depth sensing, in accordance with an embodiment.

FIG. 5 is a block diagram of a HMD system in which a console operates,in accordance with an embodiment.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION

A depth camera assembly (DCA) for determining depth information ofobjects in a local area surrounding some or all of the DCA. The DCAincludes a light source, a camera and a controller. The light sourceincludes a laser source and a modulator that generates light that is,e.g., circularly polarized at a first handedness, using light emittedfrom the laser source. The light source also projects the generatedcircularly polarized light into the local area. The camera capturesportions of the circularly polarized light reflected from the objects inthe local area. The camera is configured as a polarization sensitivecamera that detects the reflected circularly polarized light of a secondhandedness that may be opposite the first handedness. The use ofpolarized light increases a signal-to-noise ratio (SNR) as an intensityof un-polarized background ambient light can be efficiently reduced atthe polarization sensitive camera. The controller determines depthinformation based on the captured portions of the reflected circularlypolarized light.

Disclosed embodiments relate to a DCA based on a high speed circularlypolarized illumination and detection technique. An illuminator of theDCA generates circularly polarized light for scanning an environmentsurrounding some or all of the DCA. In some embodiments, for depthsensing methods based on structured light illumination, the generatedcircularly polarized light is structured light of a defined pattern,e.g., a pattern of light having parallel stripes in a propagatingdirection. In some embodiments, the illuminator of the DCA includes anacousto-optic deflector to actively scan the environment in a high speed(MHz speed) using, e.g., infrared wavelength(s). In alternateembodiments, the illuminator of the DCA includes a liquid crystal deviceto actively scan the environment in a moderate speed (kHz speed) using,e.g., infrared wavelength(s). A circularly polarized detector camerawith a large field-of-view may be utilized to detect portions of thecircularly polarized light reflected from the environment. Note that theportions of the circularly polarized light can be also scattered fromone or more objects in the environment, wherein scattering represents aform of diffuse reflection. In some embodiments, for depth sensingmethods based on time-of-flight, the circularly polarized detectorcamera includes a single pixel detector. In alternate embodiments, fordepth sensing methods based on structured light illumination, thecircularly polarized detector camera includes a two-dimensional detectorpixel array.

In some embodiments, the DCA is integrated into a head-mounted display(HMD) that captures data describing depth information in a local areasurrounding some or all of the HMD. The HMD may be part of, e.g., avirtual reality (VR) system, an augmented reality (AR) system, a mixedreality (MR) system, or some combination thereof. The HMD furtherincludes an electronic display and an optical assembly. The electronicdisplay is configured to emit image light. The optical assembly isconfigured to direct the image light to an exit pupil of the HMDcorresponding to a location of a user's eye, the image light comprisingthe depth information of the objects in the local area determined by theDCA.

FIG. 1 is a diagram of a HMD 100, in accordance with an embodiment. TheHMD 100 may be part of, e.g., a VR system, an AR system, a MR system, orsome combination thereof. In embodiments that describe AR system and/ora MR system, portions of a front side 102 of the HMD 100 are at leastpartially transparent in the visible band (˜380 nm to 750 nm), andportions of the HMD 100 that are between the front side 102 of the HMD100 and an eye of the user are at least partially transparent (e.g., apartially transparent electronic display). The HMD 100 includes a frontrigid body 105, a band 110, and a reference point 115. The HMD 100 alsoincludes a DCA configured to determine depth information of a local areasurrounding some or all of the HMID 100. The HMID 100 also includes animaging aperture 120 and an illumination aperture 125, and anillumination source of the DCA emits light (e.g., structured light)through the illumination aperture 125. An imaging device of the DCAcaptures light from the illumination source that is reflected from thelocal area through the imaging aperture 120. Light emitted from theillumination source of the DCA through the illumination aperture 125comprises polarized light, as discussed in more detail in conjunctionwith FIGS. 2-4. Light from the local area received through the imagingaperture 120 and captured by the imaging device of the DCA includesambient light and a portion of the polarized light reflected from one ormore objects in the local area. The imaging device of the DCA reduces anintensity of the received light having polarization different from aspecific polarization related to the polarized light to generatefiltered light substantially composed of light of the specificpolarization, thus increasing an SNR of the received light, as discussedin more detail in conjunction with FIGS. 2-4. The imaging device of theDCA detects the portion of the polarized light reflected from the one ormore objects in the local area having the specific polarization usingthe filtered light, as also discussed in more detail in conjunction withFIGS. 2-4.

The front rigid body 105 includes one or more electronic displayelements (not shown in FIG. 1), one or more integrated eye trackingsystems (not shown in FIG. 1), an Inertial Measurement Unit (IMU) 130,one or more position sensors 135, and the reference point 115. In theembodiment shown by FIG. 1, the position sensors 135 are located withinthe IMU 130, and neither the IMU 130 nor the position sensors 135 arevisible to a user of the HMD 100. The IMU 130 is an electronic devicethat generates fast calibration data based on measurement signalsreceived from one or more of the position sensors 135. A position sensor135 generates one or more measurement signals in response to motion ofthe HMD 100. Examples of position sensors 135 include: one or moreaccelerometers, one or more gyroscopes, one or more magnetometers,another suitable type of sensor that detects motion, a type of sensorused for error correction of the IMU 130, or some combination thereof.The position sensors 135 may be located external to the IMU 130,internal to the IMU 130, or some combination thereof.

FIG. 2 is a cross section 200 of the front rigid body 105 of the HMD 100shown in FIG. 1. As shown in FIG. 2, the front rigid body 105 includesan electronic display 210 and an optical assembly 220 that togetherprovide image light to an exit pupil 225. The exit pupil 225 is thelocation of the front rigid body 105 where a user's eye 230 ispositioned. For purposes of illustration, FIG. 2 shows a cross section200 associated with a single eye 230, but another optical assembly 220,separate from the optical assembly 220, provides altered image light toanother eye of the user. The front rigid body 105 also has an opticalaxis corresponding to a path along which image light propagates throughthe front rigid body 105.

The electronic display 210 generates image light. In some embodiments,the electronic display 210 includes an optical element that adjusts thefocus of the generated image light. The electronic display 210 displaysimages to the user in accordance with data received from a console (notshown in FIG. 2). In various embodiments, the electronic display 210 maycomprise a single electronic display or multiple electronic displays(e.g., a display for each eye of a user). Examples of the electronicdisplay 210 include: a liquid crystal display (LCD), an organic lightemitting diode (OLED) display, an inorganic light emitting diode (ILED)display, an active-matrix organic light-emitting diode (AMOLED) display,a transparent organic light emitting diode (TOLED) display, some otherdisplay, a projector, or some combination thereof. The electronicdisplay 210 may also include an aperture, a Fresnel lens, a convex lens,a concave lens, a diffractive element, a waveguide, a filter, apolarizer, a diffuser, a fiber taper, a reflective surface, a polarizingreflective surface, or any other suitable optical element that affectsthe image light emitted from the electronic display. In someembodiments, one or more of the display block optical elements may haveone or more coatings, such as anti-reflective coatings.

The optical assembly 220 magnifies received light from the electronicdisplay 210, corrects optical aberrations associated with the imagelight, and the corrected image light is presented to a user of the HMD100. At least one optical element of the optical assembly 220 may be anaperture, a Fresnel lens, a refractive lens, a reflective surface, adiffractive element, a waveguide, a filter, or any other suitableoptical element that affects the image light emitted from the electronicdisplay 210. Moreover, the optical assembly 220 may include combinationsof different optical elements. In some embodiments, one or more of theoptical elements in the optical assembly 220 may have one or morecoatings, such as anti-reflective coatings, dichroic coatings, etc.Magnification of the image light by the optical assembly 220 allowselements of the electronic display 210 to be physically smaller, weighless, and consume less power than larger displays. Additionally,magnification may increase a field-of-view of the displayed media. Forexample, the field-of-view of the displayed media is such that thedisplayed media is presented using almost all (e.g., 110 degreesdiagonal), and in some cases all, of the user's field-of-view. In someembodiments, the optical assembly 220 is designed so its effective focallength is larger than the spacing to the electronic display 210, whichmagnifies the image light projected by the electronic display 210.Additionally, in some embodiments, the amount of magnification may beadjusted by adding or removing optical elements.

As shown in FIG. 2, the front rigid body 105 further includes a DCA 240for determining depth information of one or more objects in a local area245 surrounding some or all of the HMD 100. The DCA 240 includes apolarized light generator 250, an imaging device 255, and a controller260 that may be coupled to both the polarized light generator 250 andthe imaging device 255. The polarized light generator 250 emitspolarized light through the illumination aperture 125. The polarizedlight generator 250 illuminates the local area 245 with polarized light265 in accordance with emission instructions generated by the controller260. The controller 260 may control operation of certain components ofthe polarized light generator 250, based on the emission instructions.In some embodiments, the controller 260 may provide the emissioninstructions to a modulator of the polarized light generator 250 tocontrol modulation of the polarized light 265, e.g., to controlpolarization of the polarized light 265 and/or to control afield-of-view of the local area 245 illuminated by the polarized light265. More details about controlling the modulator of the polarized lightgenerator 250 by the controller 260 are disclosed in conjunction withFIG. 3.

The polarized light generator 250 may include a plurality of emittersthat each emits light having certain characteristics (e.g., wavelength,polarization, coherence, pulse width, temporal behavior, etc.). Thecharacteristics may be the same or different between emitters, and theemitters can be operated simultaneously or individually. In oneembodiment, the plurality of emitters could be, e.g., laser diodes(e.g., edge emitters), inorganic or organic LEDs, a vertical-cavitysurface-emitting laser (VCSEL), or some other source. In someembodiments, a single emitter or a plurality of emitters in thepolarized light generator 250 can emit light having a structured lightpattern. In some embodiments, the polarized light generator 250 includesa laser diode (e.g., infrared laser diode), a modulator for modulatinglight emitted from the laser diode, and a polarizing element forgenerating polarized light, as disclosed in more detail in conjunctionwith FIG. 3.

The imaging device 255 is configured as a polarization sensitive camerathat captures, through the imaging aperture 120, portions of thepolarized light 265 reflected from the local area 245. The imagingdevice 255 may be implemented as a charge-coupled device (CCD) camera ora complementary metal-oxide-semiconductor (CMOS) camera. The imagingdevice 255 includes a polarization sensitive photodetector that uses,e.g., optically anisotropic materials to detect photons of a specificpolarization. The polarization refers to type (e.g., linear, circular,elliptical, etc.) and orientation/helicity. The imaging device 255captures one or more images of one or more objects in the local area 245illuminated with the polarized light 265. In some embodiments, for depthsensing based on time-of-flight, the imaging device 255 includes aphotodetector having one or more pixels. Each pixel in the photodetectorof the imaging device 255 may include a multiple storage bins, and theimaging device can be configured to store charge in each storage bin fora particular amount of time.

The controller 260 is configured to determine depth information for theone or more objects based at least in part on the captured portions ofthe reflected polarized light. In some embodiments, for depth sensingbased on structured light illumination, the controller 260 is configuredto determine depth information based on phase-shifted patterns of theportions of the reflected polarized light distorted by shapes of theobjects in the local area, and to use triangulation calculation toobtain a depth map of the local area. In alternate embodiments, fordepth sensing based on time-of-flight, the controller 260 is configuredto determine depth information using a ratio of charge between thestorage bins associated with each pixel in the photodetector of theimaging device 255. In some embodiments, the controller 260 provides thedetermined depth information to a console (not shown in FIG. 2) and/oran appropriate module of the HMD 100 (e.g., a varifocal module, notshown in FIG. 2). The console and/or the HMD 100 may utilize the depthinformation to, e.g., generate content for presentation on theelectronic display 210. More details about the DCA 240 that includes thepolarized light generator 250 and the imaging device 255 configured as apolarization sensitive camera are disclosed in conjunction with FIG. 3.

In some embodiments, the front rigid body 105 further comprises an eyetracking system (not shown in FIG. 2) that determines eye trackinginformation for the user's eye 230. The determined eye trackinginformation may comprise information about an orientation of the user'seye 230 in an eye-box, i.e., information about an angle of an eye-gaze.An eye-box represents a three-dimensional volume at an output of a HMIDin which the user's eye is located to receive image light. In oneembodiment, the user's eye 230 is illuminated with a structured lightpattern. Then, the eye tracking system can use locations of thereflected structured light pattern in a captured image to determine eyeposition and eye-gaze. In another embodiment, the eye tracking systemdetermines eye position and eye-gaze based on magnitudes of image lightcaptured over a plurality of time instants.

In some embodiments, the front rigid body 105 further comprises avarifocal module (not shown in FIG. 2). The varifocal module may adjustfocus of one or more images displayed on the electronic display 210,based on the eye tracking information. In one embodiment, the varifocalmodule adjusts focus of the displayed images and mitigatesvergence-accommodation conflict by adjusting a focal distance of theoptical assembly 220 based on the determined eye tracking information.In another embodiment, the varifocal module adjusts focus of thedisplayed images by performing foveated rendering of the one or moreimages based on the determined eye tracking information. In yet anotherembodiment, the varifocal module utilizes the depth information from thecontroller 260 to generate content for presentation on the electronicdisplay 210.

FIG. 3 is an example DCA 300 configured for depth sensing based onpolarized light, in accordance with an embodiment. The DCA 300 includesa polarized light generator 305, an imaging device 310, and a controller315 coupled to both the polarized light generator 305 and the imagingdevice 310. The DCA 300 may be configured to be a component of the HMD100 in FIG. 1. Thus, the DCA 300 may be an embodiment of the DCA 240 inFIG. 2; the polarized light generator 305 may be an embodiment of thepolarized light generator 250 in FIG. 2; and the imaging device 310 maybe an embodiment of the imaging device 255 in FIG. 2.

The polarized light generator 305 is configured to illuminate a localarea 320 with polarized light in accordance with emission instructionsfrom the controller 315. The polarized light generator 305 includes anillumination source 325 configured to emit one or more optical beams330. The illumination source 325 can be implemented as a light sourceoperating in a continuous wave (CW) mode or in a pulsed mode. In someembodiments, the illumination source 325 is implemented as a lightsource with high spatial coherence, such as a super luminescent diode(SLED) or a narrow band light emitting diode (LED). In some otherembodiments, the illumination source 325 is implemented as a laserdiode. In one or more embodiments, the illumination source 325 canoperate in the far infrared regime. The illumination source 325 based onan infrared laser diode can be used for depth sensing illumination inboth indoor and outdoor surroundings because a spectrum of backgroundambient light coming from sun, common lamps and lighting does notoverlap with an infrared spectrum and can be efficiently suppressed atthe imaging device 310. The infrared light emission is invisible to aneye and is far from the optical spectrum of light emitted from commonbackground light sources (e.g., for solar, typically 0.3 um to 2.5 um;for halogen lamps, typically 0.3 um to 2 um). In some embodiments, theillumination source 325 can directly generate the one or more opticalbeams 330 as polarized light, e.g., circularly polarized light. Theillumination source 325 may include a polarizing element (not shown inFIG. 3) that generates the polarized one or more optical beams 330,based in part on the emission instructions from the controller 315. Thepolarizing element may be integrated into the illumination source 325 orplaced in front of the illumination source 325. A beam conditioningassembly 335 collects light from the illumination source 325 and directsit toward a portion of a modulator 340. The beam conditioning assembly335 is composed of one or more optical elements (lenses). Theillumination source 325 (e.g., implemented as circular polarizedinfrared laser diode) can produce dynamic polarized illumination usingthe modulator 340 of various types, as discussed in more detail below.

The modulator 340 is configured to modulate the one or more opticalbeams 330 based in part on the emission instructions from the controller315 to form modulated light 345 for scanning the local area 320. Themodulator 340 may be configured to diffract the one or more opticalbeams 330 based in part on the emission instructions from the controller315 to provide a specific field-of-view for scanning the local area 320.The modulator 340 may also include a polarizing element (not shown inFIG. 3) that generates the modulated light 345 as polarized light theone or more optical beams 330, based in part on the emissioninstructions from the controller 315. In some embodiments, for depthsensing of the local area 320 based on time-of-flight, the modulator 340can be implemented as an acousto-optic deflector, an electro-opticalmodulator or a microelectromechanical system (MEM) mirror scanner,wherein the illumination source 325 can be implemented as a laser diodeoperating in the pulsed mode. In alternate embodiments, for depthsensing of the local area 320 based on structured light illumination,the modulator 340 can be implemented as diffractive-optics, anacousto-optic deflector or an electro-optical modulator, wherein theillumination source 325 can be implemented as a laser diode in the CW orpulsed mode. In this case, the modulated light 345 includes a structuredlight pattern, e.g., having parallel stipes in a propagating directionalong z axis.

The modulator 340 implemented as an acousto-optic deflector or anelectro-optical modulator (e.g., based on liquid crystal) can beemployed to directly emit the modulated light 345 as, e.g., circularlypolarized light for scanning the local area 320. In some embodiments,the electro-optical modulator 340 is implemented as a liquid crystalgrating device. In some other embodiments, an acousto-optic deflectorused as the modulator 340 can be either a bulk device or a surface wavedevice (SAW), wherein Germanium (Ge) crystal can be used as anacousto-optic interaction medium. An acousto-optic deflector used as themodulator 340 can not only dynamically deflect and scan light withunprecedented high speed, but also contains no mechanically moving parts(in contrast to MEM mirror scanners) and can be controlledelectronically, e.g., based on the emission instructions from thecontroller 315. In alternate embodiments, the modulator 340 isimplemented as a thin grating operating in the Raman-Nath regime. Themodulator 340 operating in the Raman-Nath regime can directly generatethe modulated light 345 as polarized light satisfying the Raman-Nathdiffraction condition.

The modulator 340 implemented as an acousto-optic deflector generatesthe modulated light 345 by diffracting the one or more optical beams330. In some embodiments, the modulator 340 implemented as theacousto-optic deflector is configured to function as a dynamicdiffraction grating that diffracts the one or more optical beams 330 toform the modulated light 345, based in part on the emission instructionsfrom the controller 315. The modulator 340 implemented as theacousto-optic deflector may include a transducer and a diffraction area(not shown in FIG. 3). Responsive to a radio frequency (RF) in theemission instructions, the transducer of the acousto-optic deflector canbe configured to generate a sound wave in the diffraction area of theacousto-optic deflector to form the dynamic diffraction grating. Themodulated light 345 generated by the modulator 340 implemented as theacousto-optic deflector represents structured light or hybrid structuredlight composed of patterns, e.g., a grid of laser dots. Each laser dotcan be produced by scanning the one or more optical beams 330 emittedfrom the illumination source 325 (e.g., laser diode) using a specific RFfrequency that drives the acousto-optic deflector to Bragg match theincoming one or more optical beams 330. The transducer of theacousto-optic deflector launches an acoustic wave to an anisotropiccrystal of the acousto-optic deflector that deflects the one or moreoptical beams 330 at the Bragg angle. The output polarization of themodulated light satisfying the Bragg phase matching condition can beoptimized at either right handed circular polarization or left handedcircular polarization.

As shown in FIG. 3, a projection assembly 350 is positioned in front ofthe modulator 340. The projection assembly 350 includes one or moreoptical elements (lenses). In some embodiments, when the modulated light345 is generated as un-polarized light, the projection assembly 350 alsoincludes a polarizing element 355, as shown in FIG. 3. The polarizingelement 355 polarizes the modulated light 345 to form polarized light360. The projection assembly 350 projects the polarized light 360 intothe local area 320. The polarized light 360 is circularly polarizedlight (e.g., right handed or in other embodiments left handed). Inalternate embodiments, the polarized light 360 is linearly polarizedlight (vertical and horizontal), or elliptically polarized light (rightor left). The polarizing element 355 can be a linear polarizer, acircular polarizer, an elliptical polarizer, etc. The polarizing element355 can be implemented as a thin film polarizer (absorptive,reflective), a quarter wave plate combined with a linear polarizer, etc.

The polarized light 360 illuminates portions of the local area 320,including one or more objects in the local area 320. Reflected polarizedlight 365 that propagates toward the imaging device 310 is generatedbased on reflection of the polarized light 360 from the one or moreobjects in the local area 320. Due to the reflection, polarization ofthe reflected polarized light 365 may change in comparison withpolarization of the polarized light 360. Furthermore, simultaneouslywith the reflected polarized light 365, background ambient light 370also propagates toward the imaging device 310, wherein the backgroundambient light 370 is emitted from at least one ambient light source 375,e.g., sun in an outdoor environment or at least one lamp in an outdoorenvironment. In some embodiments, the background ambient light 370reaching the imaging device 310 may be also reflected from the one ormore objects in the local area 320. The background ambient light 370 istypically un-polarized or has polarization different from polarizationof the reflected polarized light 365 generated based on reflection ofthe polarized light 360 from the one or more objects in the local area320.

The imaging device 310 includes a polarizing element 380 and a camera385. The polarizing element 380 is positioned in front of the camera385. The polarizing element 380 is configured to receive portions of thereflected polarized light 365 having a specific polarization and topropagate the received portions of reflected polarized light 365 to thecamera 385. The polarizing element 380 is also configured to block ormitigate propagation of the background ambient light 370 to the camera385. Thus, the imaging device 310 is configured to function as apolarization sensitive camera. In some embodiments, the receivedportions of the reflected polarized light 365 can be selected from, forexample, linearly polarized light (vertical and horizontal), righthanded circularly polarized light, left handed circularly polarizedlight, and elliptically polarized light (left or right handed). Thepolarizing element 380 can be a linear polarizer, a circular polarizer,an elliptical polarizer, etc. The polarizing element 380 can beimplemented as a thin film polarizer (absorptive, reflective), a quarterwave plate combined with a linear polarizer, etc. In some embodiments,the polarizing element 380 has a handedness or orientation that isopposite to a handedness/orientation of the polarizing element 355 inthe polarized light generator 305. In alternate embodiments, ahandedness/orientation of the polarizing element 380 is same as ahandedness/orientation of the polarizing element 355.

The camera 385 captures one or more images of the one or more objects inthe local area 320 by capturing the portions of the reflected polarizedlight 365 having the specific polarization. In some embodiments, thecamera 385 is an infrared camera configured to capture images in theinfrared. The camera 385 can be configured to operate with a frame ratein the range of kHz to MHz for fast detection of objects in the localarea 320. In some embodiments, the polarizing element 380 is integratedwith the camera 385. In alternate embodiments, the polarizing element380 is internal to the camera 385. In some embodiments, for depthsensing of the local area 320 based on time-of-flight, the camera 385can be implemented as a single-pixel detector. In alternate embodiments,for depth sensing of the local area 320 based on structured lightillumination, the camera 385 can be implemented as a two-dimensionalpixel array.

In some embodiments, the camera 385 is configured to have a certaininherent polarization, e.g., a circular polarization. For example, thecamera 385 having an inherent circular polarization detects portions ofthe reflected circularly polarized light 365, e.g., left handedcircularly polarized light or right handed circularly polarized lightreflected from the one or more objects in the local area 320. Uponreflection of the polarized light 360 from the one or more objects inthe local area 320, a helicity of the reflected polarized light 365 maybe changed relative to a helicity of the polarized light 360. Forexample, in the case of reflection from one or more specular objects inthe local area 320 that closely resemble mirrors having M44 coefficientof −1 in the Mueller scattering matrix, the helicity of the reflectedpolarized light 365 can be flipped relative to the helicity of thepolarized light 360.

In some embodiments, the polarized light 360 generated by the polarizedlight generator 305 illuminates one or more highly diffusing objects inthe local area 320. Temporal pulses associated with portions of thereflected polarized light 365 generated based on reflection of thepolarized light 360 from the one or more highly diffusing objects in thelocal area 320 may be detected by the camera 385, e.g., the single-pixelcamera 385. In this case, the coherence is maintained and a degree ofpolarization of the portions of the reflected polarized light 365 ishigh enough to distinguish, at the camera 385, the portions of thereflected polarized light 365 from the background ambient light 370.Therefore, for depth sensing of diffusing objects and suppressing theun-polarized background ambient light 370, the polarizing element 380can be applied in front of the camera 385, wherein the polarizingelement 380 is configured (e.g., based in part on instructions from thecontroller 315) for propagating light of a circular polarizationopposite of a circular polarization of the emitted polarized light 360.In some embodiments, the camera 385 includes a camera lens thatspatially integrates intensities of portions of the reflected polarizedlight 365 received within a specific field-of-view of the camera lens,wherein the polarizing element 380 can be configured to receive light ata specific near-infrared wavelength.

In some embodiments, the polarizing element 380 can be configured (e.g.,based in part on instructions from the controller 315) to accept andpropagate portions of the reflected polarized light 365 having a stateof polarization (SOP) of the polarized light 360 that illuminates thelocal area 320. In the same time, the polarizing element 380 is alsoconfigured (e.g., based in part on instructions from the controller 315)to block propagation of other light components, including the backgroundambient light 370, having different SOPs. Note in embodiments where thecamera 385 is a single pixel detector, a field-of-view is relativelylarger compared to a field-of-view of the imaging device 310 when thecamera 385 is based on an array of pixels. Therefore, a relativelylarger amount of the background ambient light 370 can be integrated intothe single pixel detector in comparison with the array of pixels. Thebackground ambient light 370 comes from photons generated by either anindoor room lighting device or outdoor sun light (i.e., the ambientlight source 375 shown in FIG. 3), wherein the sun light and light frommost indoor lightings are generally un-polarized after propagatingthrough the atmosphere. Thus, a particular SOP of the polarized light360 can be selected for illumination of the local area 320, e.g., basedin part on the emission instructions from the controller 315. Also, theimaging device 310 can be configured (e.g., based in part oninstructions from the controller 315) to detect portions of thereflected light 365 having the particular SOP in order to enhance an SNRwithin the field-of-view of the camera 385 (e.g., the single pixeldetector) while suppressing the ambient background light 370.

SOP of the un-polarized ambient light source 375, such as sun and indoorlightings, can be treated as a superposition of all states ofpolarizations, i.e., linear polarization, circular polarization,elliptical polarization, etc. Upon reflection of the un-polarizedbackground ambient light 370 from surfaces of one or more objects in thelocal area 320, each individual SOP in the scattered/reflectedbackground ambient light 370 can be rotated. But, as a whole, thereflected background ambient light 370 propagating toward the imagingdevice 310 remains un-polarized if the one or more objects in the localarea 320 are isotropic and non-polarizing. While a few objects cantransform the SOP, the background ambient light 370 in general remainsun-polarized after reflection because all states of polarization in thebackground ambient light 370 are transformed as a whole. In contrast, inthe case of the polarized light 360 having a single SOP, such as acircular SOP, a helicity can be flipped after reflection of thepolarized light 360 from specular objects in the local area 320. In thecase of reflection of the polarized light 360 from diffuse objects inthe local area 320, a degree of polarization can be decreased dependingon a particular diffuse object. However, the polarized light 360 havinga circular polarization exhibits a polarization memory effect meaningthat the degree of polarization may remain the same after reflection ofthe polarized light 360 from the diffuse objects in the local area 320.Another reason for generating the polarized light 360 having a circularpolarization is that a circularly polarized light is very rare innature. Therefore, for achieving efficient suppression of the backgroundambient light 370 at the imaging device 310, the local area 320 can beilluminated with the polarized light 360 having a circular polarization.Thus, the polarized light generator 305 in FIG. 3 can be implemented asa circularly polarized laser scanner having the modulator 340implemented as an acousto-optic deflector. The imaging device 310 can bethen implemented as a circularly polarized camera detector, e.g., thesingle pixel detector, operating at same wavelength(s) as the polarizedlight generator 305.

FIG. 4 is a flow chart illustrating a process 400 of determining depthinformation of objects in a local area based on polarized light, whichmay be implemented at the HMD 100 shown in FIG. 1, in accordance with anembodiment. The process 400 of FIG. 4 may be performed by the componentsof a DCA (e.g., the DCA 300). Other entities (e.g., a HMD and/orconsole) may perform some or all of the steps of the process in otherembodiments. Likewise, embodiments may include different and/oradditional steps, or perform the steps in different orders.

The DCA generates 410 (e.g., via a controller) emission instructions.The DCA may provide the emission instructions to an illumination sourceand a modulator within the DCA. Based on the emission instructions, theillumination source may emit one or more optical beams. Based on theemission instructions, the modulator may modulate the one or moreoptical beams. In some embodiments, the DCA generates the emissioninstructions which include information about a radio frequency.Responsive to the radio frequency in the emission instructions, the DCAgenerates a sound wave within the modulator to form a dynamicdiffraction grating for diffraction of the one or more optical beams.

The DCA generates 420 (e.g., via the modulator) modulated light from theone or more optical beams. In some embodiments, the DCA generates 420the modulated light by diffracting the one or more optical beams, basedin part on the emission instructions. In some embodiments, the DCAmodifies the radio frequency in the emission instructions to adjust adiffraction angle at which the one or more optical beams are diffractedand interfered to form the modulated light. The modulator may include apolarizing element implemented as a circular polarizer that generatesthe modulated light as circularly polarized light using the one or moreoptical beams, based in part on the emission instructions. In someembodiments, the modulator may directly generate the modulated light ascircularly polarized light satisfying the Bragg phase matchingcondition.

The DCA projects 430 (e.g., via a projection assembly) the modulatedlight into a local area as polarized light having a first polarization.The polarized light may be circularly polarized at a first handedness.In some embodiments, the DCA projects 440 the polarized light toilluminate a wide field-of-view of the local area for accurate depthsensing of the local area. The DCA may also control (e.g., via thecontroller) a size of the illuminated portion of the local area bycontrolling the dynamic diffraction grating within the DCA.

The DCA receives 440 (e.g., via an imaging device) light from the localarea, the received light including ambient light and a portion of thepolarized light reflected from one or more objects in the local area. AnSNR of the received light may be below a threshold for accuratedetection of the portion of the reflected polarized light due to a highlevel of intensity of the ambient light. The ambient light may beemitted from at least one ambient light source, e.g., sun in an outdoorenvironment or at least one lamp in an outdoor environment.

The DCA reduces 450 (e.g., via the imaging device) an intensity of thereceived light having polarization different from a second polarizationto generate filtered light substantially composed of light of the secondpolarization. In some embodiments, the second polarization is orthogonalto the first polarization of the emitted polarized light. In alternateembodiments, the second polarization is same as the first polarizationof the emitted polarized light. In some embodiments, the imaging deviceof the DCA includes a polarizing element and a camera, wherein thepolarizing element is positioned in front of the camera. The polarizingelement may be configured to propagate the portion of the polarizedlight reflected from the local area having the second polarization andto block propagation of other light components (e.g., the ambient light)having polarization information different from the second polarization.The portion of the reflected polarized light may be circularly polarizedat a second handedness, which may be different from the first handednessof the emitted polarized light.

The DCA detects 460 (e.g., via the imaging device) the portion of thepolarized light reflected from the one or more objects in the local areahaving the second polarization using the filtered light. In someembodiments, the polarizing element of the imaging device is configuredto propagate to the camera the portion of the reflected polarized lighthaving the second polarization. In the same time, the polarizing elementof the imaging device is configured to block propagation of light havingpolarization different from the second polarization.

The DCA determines 470 (e.g., via the controller) depth information forthe one or more objects based at least in part on the detected portionof the reflected polarized light. In some embodiments, for depth sensingbased on structured light illumination, the DCA captures phase-shiftedpatterns of the portion of the reflected polarized light distorted byshapes of the objects in the local area, and uses triangulationcalculation to obtain a depth map of the local area. The DCA may enhancedepth resolution of the local area based on information about the secondpolarization of the detected portion of the reflected polarized light.In alternate embodiments, for depth sensing based on time-of-flight, theDCA determines the depth information using a ratio of charges stored instorage bins associated with each pixel in a photodetector of theimaging device. In this case, the imaging device can be configured tostore charge in each storage bin associated with an intensity ofreceived light for a particular amount of time.

In some embodiments, the DCA is configured as part of a HMD, e.g., theHMD 100 in FIG. 1. In one embodiment, the DCA provides the determineddepth information to a console coupled to the HMD. The console is thenconfigured to generate content for presentation on an electronic displayof the HMD, based on the depth information. In another embodiment, theDCA provides the determined depth information to a module of the HMDthat generates content for presentation on the electronic display of theHMID, based on the depth information. In an alternate embodiment, theDCA is integrated into a HMID as part of an AR system. In this case, theDCA may be configured to sense and display objects behind a head of auser wearing the HMD or display objects recorded previously.

In some embodiments, the DCA is configured as part of AR glasses or MRglasses, where the DCA is implemented as a form of integrated optics andcircuits. The DCA may include a polarized light generator implemented asan on-chip active polarized illumination device. Additionally, the DCAmay include an imaging device implemented as an on-chip efficientpolarization sensitive sensor device.

System Environment

FIG. 5 is a block diagram of one embodiment of a HMD system 500 in whicha console 510 operates. The HMID system 500 may operate in a VR systemenvironment, an AR system environment, a MR system environment, or somecombination thereof. The HMID system 500 shown by FIG. 5 comprises a HMD505 and an input/output (I/O) interface 515 that is coupled to theconsole 510. While FIG. 5 shows an example HMD system 500 including oneHMD 505 and on I/O interface 515, in other embodiments any number ofthese components may be included in the HMD system 500. For example,there may be multiple HMDs 505 each having an associated I/O interface515, with each HMD 505 and I/O interface 515 communicating with theconsole 510. In alternative configurations, different and/or additionalcomponents may be included in the HMD system 500. Additionally,functionality described in conjunction with one or more of thecomponents shown in FIG. 5 may be distributed among the components in adifferent manner than described in conjunction with FIG. 5 in someembodiments. For example, some or all of the functionality of theconsole 510 is provided by the HMD 505.

The HMD 505 is a head-mounted display that presents content to a usercomprising virtual and/or augmented views of a physical, real-worldenvironment with computer-generated elements (e.g., two-dimensional (2D)or three-dimensional (3D) images, 2D or 3D video, sound, etc.). In someembodiments, the presented content includes audio that is presented viaan external device (e.g., speakers and/or headphones) that receivesaudio information from the HMD 505, the console 510, or both, andpresents audio data based on the audio information. The HMD 505 maycomprise one or more rigid bodies, which may be rigidly or non-rigidlycoupled together. A rigid coupling between rigid bodies causes thecoupled rigid bodies to act as a single rigid entity. In contrast, anon-rigid coupling between rigid bodies allows the rigid bodies to moverelative to each other. An embodiment of the HMD 505 is the HMD 100described above in conjunction with FIG. 1.

The HMD 505 includes a DCA 520, an electronic display 525, an opticalassembly 530, one or more position sensors 535, an IMU 540, an optionaleye tracking system 545, and an optional varifocal module 550. Someembodiments of the HMD 505 have different components than thosedescribed in conjunction with FIG. 5. Additionally, the functionalityprovided by various components described in conjunction with FIG. 5 maybe differently distributed among the components of the HMD 505 in otherembodiments.

The DCA 520 captures data describing depth information of a local areasurrounding some or all of the HMD 505. The DCA 520 can compute thedepth information using the data (e.g., based on captured portions ofpolarized light), or the DCA 520 can send this information to anotherdevice such as the console 510 that can determine the depth informationusing the data from the DCA 520.

The DCA 520 includes a polarized light generator, an imaging device anda controller. The polarized light generator of the DCA 520 is configuredto illuminate the local area with polarized light in accordance withemission instructions. The polarized light generator comprises anillumination source, a modulator, and a projection assembly. Theillumination source is configured to emit one or more optical beams. Insome embodiments, the illumination source directly generates the one ormore optical beams as light of a certain polarization, e.g., based on apolarizing element integrated into the illumination source or placed infront of the illumination source. The modulator is configured tomodulate the one or more optical beams based in part on the emissioninstructions to form a modulated light for scanning the local area. Insome embodiments, the modulator is configured as a diffractive opticalelement that diffracts the one or more optical beams based in part onthe emission instructions to generate a diffracted light for scanningthe local area with a wide field-of-view. In some embodiments, themodulator includes a polarizing element for generating the modulatedlight as the polarized light using the one or more optical beams, basedin part on the emission instructions. The projection assembly isconfigured to project the polarized light into the local area. In someembodiments, the projection assembly includes the polarizing element forgenerating the polarized light using the modulated light generated bythe modulator as un-polarized light. The polarized light projected intothe local area has a first polarization, e.g., circular polarization.The imaging device of the DCA 520 is configured to capture portions ofthe polarized light reflected from the one or more objects in the localarea. The imaging device includes another polarizing element and adetector. The other polarizing element of the imaging device isconfigured to receive light from the local area, the received lightincluding ambient light and a portion of the polarized light reflectedfrom the one or more objects in the local area. The other polarizingelement of the imaging device is also configured to reduce an intensityof the received light having polarization different from a secondpolarization (e.g., the ambient light) to generate filtered lightsubstantially composed of light of the second polarization. In someembodiments, the polarizing element blocks the ambient light forreaching the detector. The detector is configured to detect the portionof the polarized light reflected from the one or more objects in thelocal area having the second polarization using the filtered light. Thecontroller of the DCA 520 may be coupled to both the polarized lightgenerator and the imaging device. The controller generates the emissioninstructions and provides the emission instructions to the polarizedlight generator. The controller is also configured to determine depthinformation for the one or more objects based at least in part on thedetected portion of the reflected polarized light. The DCA 520 is anembodiment of the DCA 240 in FIG. 2 or the DCA 300 in FIG. 3A.

The electronic display 525 displays 2D or 3D images to the user inaccordance with data received from the console 510. In variousembodiments, the electronic display 525 comprises a single electronicdisplay or multiple electronic displays (e.g., a display for each eye ofa user). Examples of the electronic display 525 include: a liquidcrystal display (LCD), an organic light emitting diode (OLED) display,an inorganic light emitting diode (ILED) display, an active-matrixorganic light-emitting diode (AMOLED) display, a transparent organiclight emitting diode (TOLED) display, some other display, or somecombination thereof.

The optical assembly 530 magnifies image light received from theelectronic display 525, corrects optical errors associated with theimage light, and presents the corrected image light to a user of the HMD505. The optical assembly 530 includes a plurality of optical elements.Example optical elements included in the optical assembly 530 include:an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, areflecting surface, or any other suitable optical element that affectsimage light. Moreover, the optical assembly 530 may include combinationsof different optical elements. In some embodiments, one or more of theoptical elements in the optical assembly 530 may have one or morecoatings, such as partially reflective or anti-reflective coatings.

Magnification and focusing of the image light by the optical assembly530 allows the electronic display 525 to be physically smaller, weighless and consume less power than larger displays. Additionally,magnification may increase the field of view of the content presented bythe electronic display 525. For example, the field of view of thedisplayed content is such that the displayed content is presented usingalmost all (e.g., approximately 110 degrees diagonal), and in some casesall, of the user's field of view. Additionally in some embodiments, theamount of magnification may be adjusted by adding or removing opticalelements.

In some embodiments, the optical assembly 530 may be designed to correctone or more types of optical error. Examples of optical error includebarrel or pincushion distortions, longitudinal chromatic aberrations, ortransverse chromatic aberrations. Other types of optical errors mayfurther include spherical aberrations, chromatic aberrations or errorsdue to the lens field curvature, astigmatisms, or any other type ofoptical error. In some embodiments, content provided to the electronicdisplay 525 for display is pre-distorted, and the optical assembly 530corrects the distortion when it receives image light from the electronicdisplay 525 generated based on the content.

The IMU 540 is an electronic device that generates data indicating aposition of the HMD 505 based on measurement signals received from oneor more of the position sensors 535 and from depth information receivedfrom the DCA 520. A position sensor 535 generates one or moremeasurement signals in response to motion of the HMD 505. Examples ofposition sensors 535 include: one or more accelerometers, one or moregyroscopes, one or more magnetometers, another suitable type of sensorthat detects motion, a type of sensor used for error correction of theIMU 540, or some combination thereof. The position sensors 535 may belocated external to the IMU 540, internal to the IMU 540, or somecombination thereof.

Based on the one or more measurement signals from one or more positionsensors 535, the IMU 540 generates data indicating an estimated currentposition of the HMD 505 relative to an initial position of the HMD 505.For example, the position sensors 535 include multiple accelerometers tomeasure translational motion (forward/back, up/down, left/right) andmultiple gyroscopes to measure rotational motion (e.g., pitch, yaw,roll). In some embodiments, the IMU 540 rapidly samples the measurementsignals and calculates the estimated current position of the HMD 505from the sampled data. For example, the IMU 540 integrates themeasurement signals received from the accelerometers over time toestimate a velocity vector and integrates the velocity vector over timeto determine an estimated current position of a reference point on theHMD 505. Alternatively, the IMU 540 provides the sampled measurementsignals to the console 510, which interprets the data to reduce error.The reference point is a point that may be used to describe the positionof the HMD 505. The reference point may generally be defined as a pointin space or a position related to the HMD's 505 orientation andposition.

The IMU 540 receives one or more parameters from the console 510. Theone or more parameters are used to maintain tracking of the HMD 505.Based on a received parameter, the IMU 540 may adjust one or more IMUparameters (e.g., sample rate). In some embodiments, certain parameterscause the IMU 540 to update an initial position of the reference pointso it corresponds to a next position of the reference point. Updatingthe initial position of the reference point as the next calibratedposition of the reference point helps reduce accumulated errorassociated with the current position estimated the IMU 540. Theaccumulated error, also referred to as drift error, causes the estimatedposition of the reference point to “drift” away from the actual positionof the reference point over time. In some embodiments of the HMD 505,the IMU 540 may be a dedicated hardware component. In other embodiments,the IMU 540 may be a software component implemented in one or moreprocessors.

In some embodiments, the eye tracking system 545 is integrated into theHMD 505. The eye tracking system 545 determines eye tracking informationassociated with an eye of a user wearing the HMD 505. The eye trackinginformation determined by the eye tracking system 545 may compriseinformation about an orientation of the user's eye, i.e., informationabout an angle of an eye-gaze. In some embodiments, the eye trackingsystem 545 is integrated into the optical assembly 530. An embodiment ofthe eye-tracking system 545 may comprise an illumination source and animaging device (camera).

In some embodiments, the varifocal module 550 is further integrated intothe HMD 505. The varifocal module 550 may be coupled to the eye trackingsystem 545 to obtain eye tracking information determined by the eyetracking system 545. The varifocal module 550 may be configured toadjust focus of one or more images displayed on the electronic display525, based on the determined eye tracking information obtained from theeye tracking system 545. In this way, the verifocal module 550 canmitigate vergence-accommodation conflict in relation to image light. Thevarifocal module 550 can be interfaced (e.g., either mechanically orelectrically) with at least one of the electronic display 525 and atleast one optical element of the optical assembly 530. Then, thevarifocal module 550 may be configured to adjust focus of the one ormore images displayed on the electronic display 525 by adjustingposition of at least one of the electronic display 525 and the at leastone optical element of the optical assembly 530, based on the determinedeye tracking information obtained from the eye tracking system 545. Byadjusting the position, the varifocal module 550 varies focus of imagelight output from the electronic display 525 towards the user's eye. Thevarifocal module 550 may be also configured to adjust resolution of theimages displayed on the electronic display 525 by performing foveatedrendering of the displayed images, based at least in part on thedetermined eye tracking information obtained from the eye trackingsystem 545. In this case, the varifocal module 550 provides appropriateimage signals to the electronic display 525. The varifocal module 550provides image signals with a maximum pixel density for the electronicdisplay 525 only in a foveal region of the user's eye-gaze, whileproviding image signals with lower pixel densities in other regions ofthe electronic display 525. In one embodiment, the varifocal module 550may utilize the depth information obtained by the DCA 520 to, e.g.,generate content for presentation on the electronic display 525.

The I/O interface 515 is a device that allows a user to send actionrequests and receive responses from the console 510. An action requestis a request to perform a particular action. For example, an actionrequest may be an instruction to start or end capture of image or videodata or an instruction to perform a particular action within anapplication. The I/O interface 515 may include one or more inputdevices. Example input devices include: a keyboard, a mouse, a gamecontroller, or any other suitable device for receiving action requestsand communicating the action requests to the console 510. An actionrequest received by the I/O interface 515 is communicated to the console510, which performs an action corresponding to the action request. Insome embodiments, the I/O interface 515 includes an IMU 540 thatcaptures calibration data indicating an estimated position of the I/Ointerface 515 relative to an initial position of the I/O interface 515.In some embodiments, the I/O interface 515 may provide haptic feedbackto the user in accordance with instructions received from the console510. For example, haptic feedback is provided when an action request isreceived, or the console 510 communicates instructions to the I/Ointerface 515 causing the I/O interface 515 to generate haptic feedbackwhen the console 510 performs an action.

The console 510 provides content to the HMD 505 for processing inaccordance with information received from one or more of: the DCA 520,the HMD 505, and the I/O interface 515. In the example shown in FIG. 5,the console 510 includes an application store 555, a tracking module560, and an engine 565. Some embodiments of the console 510 havedifferent modules or components than those described in conjunction withFIG. 5. Similarly, the functions further described below may bedistributed among components of the console 510 in a different mannerthan described in conjunction with FIG. 5.

The application store 555 stores one or more applications for executionby the console 510. An application is a group of instructions, that whenexecuted by a processor, generates content for presentation to the user.Content generated by an application may be in response to inputsreceived from the user via movement of the HMD 505 or the I/O interface515. Examples of applications include: gaming applications, conferencingapplications, video playback applications, or other suitableapplications.

The tracking module 560 calibrates the HMD system 500 using one or morecalibration parameters and may adjust one or more calibration parametersto reduce error in determination of the position of the HMD 505 or ofthe I/O interface 515. For example, the tracking module 560 communicatesa calibration parameter to the DCA 520 to adjust the focus of the DCA520 to more accurately determine positions of structured light elementscaptured by the DCA 520. Calibration performed by the tracking module560 also accounts for information received from the IMU 540 in the HMD505 and/or an IMU 540 included in the I/O interface 515. Additionally,if tracking of the HMD 505 is lost (e.g., the DCA 520 loses line ofsight of at least a threshold number of structured light elements), thetracking module 560 may re-calibrate some or all of the HMD system 500.

The tracking module 560 tracks movements of the HMD 505 or of the I/Ointerface 515 using information from the DCA 520, the one or moreposition sensors 535, the IMU 540 or some combination thereof. Forexample, the tracking module 550 determines a position of a referencepoint of the HMD 505 in a mapping of a local area based on informationfrom the HMD 505. The tracking module 560 may also determine positionsof the reference point of the HMD 505 or a reference point of the I/Ointerface 515 using data indicating a position of the HMD 505 from theIMU 540 or using data indicating a position of the I/O interface 515from an IMU 540 included in the I/O interface 515, respectively.Additionally, in some embodiments, the tracking module 560 may useportions of data indicating a position or the HMD 505 from the IMU 540as well as representations of the local area from the DCA 520 to predicta future location of the HMD 505. The tracking module 560 provides theestimated or predicted future position of the HMD 505 or the I/Ointerface 515 to the engine 555.

The engine 565 generates a 3D mapping of the area surrounding some orall of the HMD 505 (i.e., the “local area”) based on informationreceived from the HMD 505. In some embodiments, the engine 565determines depth information for the 3D mapping of the local area basedon information received from the DCA 520 that is relevant for techniquesused in computing depth. The engine 565 may calculate depth informationusing one or more techniques in computing depth from the portion of thereflected polarized light detected by the DCA 520, such as thestructured light illumination technique and the time-of-flighttechnique. In various embodiments, the engine 565 uses the depthinformation to, e.g., update a model of the local area, and generatecontent based in part on the updated model.

The engine 565 also executes applications within the HMD system 500 andreceives position information, acceleration information, velocityinformation, predicted future positions, or some combination thereof, ofthe HMD 505 from the tracking module 560. Based on the receivedinformation, the engine 565 determines content to provide to the HMD 505for presentation to the user. For example, if the received informationindicates that the user has looked to the left, the engine 565 generatescontent for the HMD 505 that mirrors the user's movement in a virtualenvironment or in an environment augmenting the local area withadditional content. Additionally, the engine 565 performs an actionwithin an application executing on the console 510 in response to anaction request received from the I/O interface 515 and provides feedbackto the user that the action was performed. The provided feedback may bevisual or audible feedback via the HMD 505 or haptic feedback via theI/O interface 515.

In some embodiments, based on the eye tracking information (e.g.,orientation of the user's eye) received from the eye tracking system545, the engine 565 determines resolution of the content provided to theHMD 505 for presentation to the user on the electronic display 525. Theengine 565 provides the content to the HMD 605 having a maximum pixelresolution on the electronic display 525 in a foveal region of theuser's gaze, whereas the engine 565 provides a lower pixel resolution inother regions of the electronic display 525, thus achieving less powerconsumption at the HMD 505 and saving computing cycles of the console510 without compromising a visual experience of the user. In someembodiments, the engine 565 can further use the eye tracking informationto adjust where objects are displayed on the electronic display 525 toprevent vergence-accommodation conflict.

Additional Configuration Information

The foregoing description of the embodiments of the disclosure has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the disclosure be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

What is claimed is:
 1. A light generator comprising: an illuminationsource configured to emit one or more optical beams; an acousto-opticdevice configured to function as a dynamic diffraction grating thatdiffracts the one or more optical beams to form modulated light based inpart on emission instructions; a controller configured to: generate theemission instructions comprising information about a radio frequency atwhich the acousto-optic device is driven, and modify the radio frequencyto adjust a diffraction angle at which the one or more optical beams arediffracted by the acousto-optic device to form the modulated light; anda projection assembly configured to project the modulated light into alocal area as polarized light having a first polarization.
 2. The lightgenerator of claim 1, wherein the acousto-optic device includes atransducer and a diffraction area, and responsive to the radio frequencyin the emission instructions, the transducer is configured to generatean acoustic wave in the diffraction area to form the dynamic diffractiongrating.
 3. The light generator of claim 1, wherein the acousto-opticdevice is further configured to directly generate the modulated lighthaving the first polarization.
 4. The light generator of claim 1,wherein the acousto-optic device is further configured to: directlygenerate the modulated light as the polarized light satisfying the Braggphase matching condition for Bragg matching the one or more opticalbeams incident to the acousto-optic device.
 5. The light generator ofclaim 1, wherein the modulated light comprises a structured lightpattern, and the acousto-optic device is further configured to: scan theone or more optical beams using a specific radio frequency that drivesthe acousto-optic device to generate a corresponding portion of thestructured light pattern.
 6. The light generator of claim 1, wherein thepolarized light is selected from a group consisting of: linearlypolarized light, circularly polarized light, and elliptically polarizedlight.
 7. The light generator of claim 1, wherein the illuminationsource generates the one or more optical beams as light of the firstpolarization by using a polarizing element associated with theillumination source.
 8. The light generator of claim 1, wherein theillumination source is selected from a group consisting of a lightsource operating in a continuous wave mode, and a light source operatingin a pulsed mode.
 9. The light generator of claim 1, wherein the lightgenerator is a component of a depth camera assembly.
 10. The lightgenerator of claim 9, wherein the depth camera assembly is integratedinto a head-mounted display.
 11. A method comprising: instructing anillumination source to emit one or more optical beams; diffracting theone or more optical beams using an acousto-optic device configured tofunction as a dynamic diffraction grating to form modulated light basedin part on emission instructions; generating the emission instructionscomprising information about a radio frequency at which theacousto-optic device is driven: modifying the radio frequency to adjusta diffraction angle at which the one or more optical beams arediffracted by the acousto-optic device to form the modulated light; andprojecting the modulated light into a local area as polarized lighthaving a first polarization.
 12. The method of claim 11, furthercomprising: responsive to the radio frequency in the emissioninstructions, generating an acoustic wave in a diffraction area of theacousto-optic device to form the dynamic diffraction grating.
 13. Themethod of claim 11, further comprising: directly generating themodulated light having the first polarization using the acousto-opticdevice.
 14. The method of claim 11, further comprising: directlygenerating the modulated light as the polarized light satisfying theBragg phase matching condition for Bragg matching the one or moreoptical beams incident to the acousto-optic device.
 15. The method ofclaim 11, wherein the modulated light comprises a structured lightpattern, and the method further comprising: driving the acousto-opticdevice using a specific radio frequency to scan the one or more opticalbeams by the acousto-optic device for generating a corresponding portionof the structured light pattern.
 16. The method of claim 11, wherein thepolarized light is selected from a group consisting of: linearlypolarized light, circularly polarized light, and elliptically polarizedlight.
 17. A depth camera assembly (DCA) comprising: an illuminationsource configured to emit one or more optical beams; an acousto-opticdevice configured to function as a dynamic diffraction grating thatdiffracts the one or more optical beams to form modulated light based inpart on emission instructions; a projection assembly configured toproject the modulated light into a local area as polarized light havinga first polarization; an imaging device configured to: receive lightfrom the local area, the received light including ambient light and aportion of the polarized light reflected from one or more objects in thelocal area, reduce an intensity of a portion of the received lighthaving a polarization different than a second polarization to generatefiltered light substantially composed of light of the secondpolarization, and detect the portion of the reflected polarized lighthaving the second polarization using the filtered light; and acontroller configured to determine depth information for the one or moreobjects based at least in part on the detected portion of the reflectedpolarized light.
 18. The DCA of claim 17, wherein the controller isfurther configured to: generate the emission instructions comprisinginformation about a radio frequency at which the acousto-optic device isdriven; and modify the radio frequency to adjust a diffraction angle atwhich the one or more optical beams are diffracted by the acousto-opticdevice to form the modulated light.